Image normalization increasing robustness of machine learning applications for medical images

ABSTRACT

A computer program, a system and a method for normalizing medical images from a type of image acquisition device using a machine learning unit are disclosed. An embodiment of the method includes receiving a set of image data with images; decomposing each of the images of the set of images into components by incorporating at least information from different settings of the image acquisition device-specific image processing algorithms; and normalizing each of the components via a machine learning unit by processing at least information from the different settings of the image acquisition device-specific processing algorithms to provide a set of normalized images with a relatively decreased variability score.

PRIORITY STATEMENT

The present application hereby claims priority under 35 U.S.C. § 119 toEuropean patent application number EP20166951.2 filed Mar. 31, 2020, theentire contents of which are hereby incorporated herein by reference.

FIELD

Example embodiments of the invention generally relate to imageprocessing and in particular to medical imaging.

BACKGROUND

Medical image processing provides support for doctors when evaluatingmedical images taken from patients. Computerized systems assist doctorsand/or medical professionals in making decisions on the presence ofdisease.

Image processing is meanwhile a typical field for Machine Learning (ML).The progress in processing speed of computers, memory capacity etc. andthe possibility to collect and process huge amounts of digital data leadto major improvements in Machine Learning. In general, Machine Learningis the use of computers (machines) in terms of hardware and software inorder to make predictions from huge data sets. Machine learningalgorithms are usually computer programs. One major difference to theclassical software development consists in that the Machine Learningsystem discloses (learns) relationships between input and output datafrom data sets. Such relationships may not be known in advance. How thedata are processed can change over the time. That is considered to bethe learning process.

Neural networks are among others a tool for machine learning. If theneural network consists of several layers, comprising a large number ofneurons, it usually allows Deep Learning. One type of neural networks isthe convolutional neural network. It comprises different types of layersincluding convolutional layers. As a result of mathematical processeskey features of images are emphasized by convolution.

The ML system is trained by specific training data sets. Test data setsare used to validate the training results. Once the system is trainedand tested sufficiently it can be considered for real-world deployment,in particular in the medical domain. The quality of the data sets usedplay a major role in the efficiency of the ML system.

Medical image data of patients will be provided to the ML system whichwill suggest diagnoses to the doctor. Reliability of diagnosis isimproved with more accurate data.

Medical imaging is typically influenced by three types of variabilities:

1) The acquisition depends on the hard- and software configuration ofthe recording device.

2) The shape of the anatomical structures varies with the relativepositioning and characteristics (e.g., age, gender) of the patient.

3) The recorded information (e.g., photon energy) is processed toproduce a vendor- and application-specific representation, which isadjustable to the demands of the examining doctor.

A first method is image impression conversion. In the literature severalmethods for transformations between image impressions have beenreported.

For instance, US2012/0321151 introduces a method for conversion of imageimpression to enable current and prior image comparison in mammography.An input image is decomposed into high-pass images and a low-pass image.Adjusting values are determined and applied to the high-pass images andthe low-pass image. Finally, an adjusted image is created. However, themethod is designed for visual image comparison and not intended to beused for optimization of machine learning training.

A second method is image normalization for machine learning. Imagenormalization is normally understood as the change of the intensityvalues in an image. Evaluation of medical images requires a specificimage quality in order to find diseases based on images comparable froma technological perspective.

Normalization of medical images has been addressed in variouspublications. For instance, researches of Phillips Healthcareinvestigated several methods for image normalization in [M. S. Vidya etal.: “Local and Global Transformations to Improve Learning of Medical,Images Applied to Chest Radiographs”, SPIE 2019v]. Well-known global(e.g., histogram-based equalization) and spatially local (e.g., adaptivehistogram equalization) methods were applied to chest X-ray images.These transformations are well-known techniques for RGB images and havebeen directly applied to the input image, without decomposition. Thepre-processed images are input for the DenseNet-121 Classifier. However,the pre-processed images are not optimized to explicitly address thevariability produced by different image impressions, as mentioned aboveunder item 3) “variability”.

Further examples for global histogram normalization are stretchingmethods which aim at reducing the impact of histogram outliers on the MLalgorithm accuracy (e.g., [S. Guendel et al.: “Learning to RecognizeAbnormalities in Chest X-Rays with Location-Aware Dense Networks”, CIARP2018]).

A pre-processing method that applies a Laplacian image decomposition andenergy normalization with subsequent image composition is described in[L. H. Neath: “CheXNet2: End-to-end Improvements for Chest DiseaseClassification”,https://cs230.stanford.edu/projects_spring_2018/reports/8290537.pdf].The pre-processed medical images improved the classification performanceof the neural network. Nevertheless, the introduced component-wiseenergy normalization does not explicitly address variability 3) producedby different configurations of image acquisition device-specific imageprocessing techniques.

SUMMARY

The application of ML techniques to medical image interpretation isgenerally based on the assumption that all sources of imagevariabilities 1) to 3) are captured by the training data. However, theinventors have discovered that this is often impractical and leads tothe well-known problem that mismatches between training andtesting/deployment (real world) conditions might produce unpredictableresults. As particularly severe clinical consequence for variability 3),the assessment of the same recorded information could be dependent onthe configuration of the image processing technique: Different imageimpressions (e.g., characterized by specific contrast, sharpness andbrightness) can result from the alternation of physical parameters withdifferent settings of an image acquisition device-specific processingtechnique, e.g. in radiography, ultrasound, computed tomography,magnetic resonance imaging.

Therefore, a problem that an embodiment of this invention aims toaddress is that the variability, mentioned above under item 3), causedby processing the recorded information with different image impressionscan lead to a lower accuracy when unaccounted for in (deep) machinelearning.

Based on the above mentioned drawbacks of the state of the art inpre-processing of images and their application for machine learning, atleast one embodiment of the present invention provides a solution whichincreases the robustness of machine learning algorithmic task. Inparticular, the training of a neural network should be improved forimages resulting from the same type of modality or more general imageacquisition device but from different image processing techniques.Finally, variabilities in image impression or image impressiondeviations should be reduced, which are caused by image acquisitiondevice-specific image processing algorithms.

Embodiments are directed to a method, respective computer programproduct and system. Advantageous aspects, features and embodiments aredescribed in the dependent claims and in the following descriptiontogether with advantages.

In the following, at least one embodiment of the invention is describedwith respect to the method as well as with respect to the system.Features, advantages or alternative embodiments which are described orclaimed in relation to one claim category can also be assigned to theother claim categories (e.g. the computer program or a computer programproduct) as well and vice versa. In other words, the subject matteraccording to the claims of the system can be improved with featuresdescribed or claimed in the context of the method. In this case, thefunctional features of the method are embodied by structural units ofthe system (processing units, e.g., normalizing is executed by anormalization unit) and vice versa, respectively.

According to a first embodiment, the present invention refers to acomputer-implemented method (i.e. executed on a processing unit of acomputer, which may e.g. be deployed in an image acquisition orprocessing apparatus) for normalizing medical images. In a preparationphase, a type of image acquisition device may be set for which themethod should be applied (e.g. X-ray, MRI, CT etc.). Thus, the imagesstem from a predetermined type of image acquisition device (in themedical domain also called ‘modality’). The method uses a machinelearning unit and comprises:

receiving a set of image data with images, in particular medical imagesfrom the predetermined type of image acquisition device, at adecomposition unit, wherein the image data have been generated by beingconverted from detector signals, acquired at a detector of an imageacquisition device (e.g. medical modality), wherein the detector signalsare converted by using or applying different settings or configurationsof the image acquisition device-specific processing algorithms;

at a decomposition unit: Decomposing each of the images of the set ofimages into components by incorporating at least information from thedifferent settings of the (applied) image acquisition device-specificprocessing algorithms; and at a normalizing unit: Normalizing each ofthe components via a machine learning unit by processing at leastinformation from the different settings of the image acquisitiondevice-specific processing algorithms to provide a set of normalizedimages as output with a decreased variability score.

In another embodiment, the invention relates to a system for normalizingmedical or non-medical images from a predetermined type of imageacquisition device using a machine learning unit. The system iscomputer-based and comprises:

a decomposition unit with an image input interface for receiving a setof image data with images, wherein the image data have been generated bybeing converted from detector signals, acquired at a detector of animage acquisition device, wherein the detector signals are converted byapplying different settings of configurations in the image acquisitiondevice-specific image processing algorithms;

wherein, the decomposition unit, incorporates at least information fromthe different settings in the modality-specific image processingalgorithms and is adapted for decomposing each of the images of the setof images into components; and

a normalizing unit, which is adapted for normalizing each of thecomponents via a machine learning unit by processing at leastinformation from the different settings of the image acquisitiondevice-specific image processing algorithms to provide a set ofnormalized images with a decreased variability score.

In another embodiment, the invention relates to a computer programproduct comprising a computer program, the computer program beingloadable into a memory unit of a computing unit, including program codesections to make the computing unit execute the method for imagenormalization according to an embodiment of the invention, when thecomputer program is executed in the computing unit.

In another embodiment, the invention relates to a computer-readablemedium, on which program code sections of a computer program are storedor saved, the program code sections being loadable into and/orexecutable in a computing unit to make the computing unit execute themethod for image normalization according to an embodiment of theinvention, when the program code sections are executed in the computingunit.

In another embodiment, the invention relates to a computer program, thecomputer program being loadable into a memory unit of a computer, inparticular, of an image processing or image acquisition device.

In another embodiment, the invention relates to a computer-implementedmethod for normalizing images from a type of image acquisition deviceusing a machine learning unit, the method comprising:

receiving a set of image data with images, the image data having beengenerated by being converted from detector signals, acquired at adetector of an image acquisition device, wherein the detector signalswere converted by applying different settings in image acquisitiondevice-specific processing algorithms;

decomposing each of the images of the set of image data into components,by incorporating at least information from the different settings of theimage acquisition device-specific processing algorithms; and

normalizing each of the components via the machine learning unit toproduce normalized components, by processing at least information fromthe different settings of the image acquisition device-specificprocessing algorithms to provide a set of normalized images with arelatively decreased variability score.

In another embodiment, the invention relates to a system for normalizingmedical images from a type of image acquisition device using a machinelearning unit, comprising:

a decomposition unit including an image input interface,

to receive a set of image data with images, the image data having beengenerated by being converted from detector signals, acquired at adetector of an image acquisition device, the detector signals havingbeen converted by applying different settings of image acquisitiondevice-specific processing algorithms, and

adapted to decompose each of the images of the set of images intocomponents by incorporating at least information from the differentsettings of the image acquisition device-specific processing algorithms;and

a normalizing unit, adapted to normalize each of the components via amachine learning unit, by processing at least information from thedifferent settings of the image acquisition device-specific processingalgorithms to provide a set of normalized images with a relativelydecreased variability score.

In another embodiment, the invention relates to a non-transitorycomputer program product storing a computer program, the computerprogram being loadable into a memory unit of a computing unit andincluding program code sections to make the computing unit execute themethod of an embodiment, when the computer program is executed in thecomputing unit.

In another embodiment, the invention relates to an image acquisitionimage acquisition device, comprising:

the system of an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter.

FIG. 1 indicates a block diagram of the general pipeline for normalizingmedical images;

FIG. 2 indicates a block diagram of an embodiment for the pipeline fornormalizing radiographical images; and

FIG. 3 shows example images with and without the normalization method

FIG. 4 is a flow chart with a sequence of method steps of thenormalizing method according to a preferred embodiment of the invention.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

The drawings are to be regarded as being schematic representations andelements illustrated in the drawings are not necessarily shown to scale.Rather, the various elements are represented such that their functionand general purpose become apparent to a person skilled in the art. Anyconnection or coupling between functional blocks, devices, components,or other physical or functional units shown in the drawings or describedherein may also be implemented by an indirect connection or coupling. Acoupling between components may also be established over a wirelessconnection. Functional blocks may be implemented in hardware, firmware,software, or a combination thereof.

Various example embodiments will now be described more fully withreference to the accompanying drawings in which only some exampleembodiments are shown. Specific structural and functional detailsdisclosed herein are merely representative for purposes of describingexample embodiments. Example embodiments, however, may be embodied invarious different forms, and should not be construed as being limited toonly the illustrated embodiments. Rather, the illustrated embodimentsare provided as examples so that this disclosure will be thorough andcomplete, and will fully convey the concepts of this disclosure to thoseskilled in the art. Accordingly, known processes, elements, andtechniques, may not be described with respect to some exampleembodiments. Unless otherwise noted, like reference characters denotelike elements throughout the attached drawings and written description,and thus descriptions will not be repeated. At least one embodiment ofthe present invention, however, may be embodied in many alternate formsand should not be construed as limited to only the example embodimentsset forth herein.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions,layers, and/or sections, these elements, components, regions, layers,and/or sections, should not be limited by these terms. These terms areonly used to distinguish one element from another. For example, a firstelement could be termed a second element, and, similarly, a secondelement could be termed a first element, without departing from thescope of example embodiments of the present invention. As used herein,the term “and/or,” includes any and all combinations of one or more ofthe associated listed items. The phrase “at least one of” has the samemeaning as “and/or”.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,”“above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or operation in addition tothe orientation depicted in the figures. For example, if the device inthe figures is turned over, elements described as “below,” “beneath,” or“under,” other elements or features would then be oriented “above” theother elements or features. Thus, the example terms “below” and “under”may encompass both an orientation of above and below. The device may beotherwise oriented (rotated 90 degrees or at other orientations) and thespatially relative descriptors used herein interpreted accordingly. Inaddition, when an element is referred to as being “between” twoelements, the element may be the only element between the two elements,or one or more other intervening elements may be present.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Incontrast, when an element is referred to as being “directly” connected,engaged, interfaced, or coupled to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between,” versus “directly between,” “adjacent,” versus“directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the invention. As used herein, the singular forms “a,”“an,” and “the,” are intended to include the plural forms as well,unless the context clearly indicates otherwise. As used herein, theterms “and/or” and “at least one of” include any and all combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “comprises,” “comprising,” “includes,” and/or“including,” when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist. Also, the term “example” is intended to refer to an example orillustration.

When an element is referred to as being “on,” “connected to,” “coupledto,” or “adjacent to,” another element, the element may be directly on,connected to, coupled to, or adjacent to, the other element, or one ormore other intervening elements may be present. In contrast, when anelement is referred to as being “directly on,” “directly connected to,”“directly coupled to,” or “immediately adjacent to,” another elementthere are no intervening elements present.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, e.g., those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Before discussing example embodiments in more detail, it is noted thatsome example embodiments may be described with reference to acts andsymbolic representations of operations (e.g., in the form of flowcharts, flow diagrams, data flow diagrams, structure diagrams, blockdiagrams, etc.) that may be implemented in conjunction with units and/ordevices discussed in more detail below. Although discussed in aparticularly manner, a function or operation specified in a specificblock may be performed differently from the flow specified in aflowchart, flow diagram, etc. For example, functions or operationsillustrated as being performed serially in two consecutive blocks mayactually be performed simultaneously, or in some cases be performed inreverse order. Although the flowcharts describe the operations assequential processes, many of the operations may be performed inparallel, concurrently or simultaneously. In addition, the order ofoperations may be re-arranged. The processes may be terminated whentheir operations are completed, but may also have additional steps notincluded in the figure. The processes may correspond to methods,functions, procedures, subroutines, subprograms, etc.

Specific structural and functional details disclosed herein are merelyrepresentative for purposes of describing example embodiments of thepresent invention. This invention may, however, be embodied in manyalternate forms and should not be construed as limited to only theembodiments set forth herein.

Units and/or devices according to one or more example embodiments may beimplemented using hardware, software, and/or a combination thereof. Forexample, hardware devices may be implemented using processing circuitrysuch as, but not limited to, a processor, Central Processing Unit (CPU),a controller, an arithmetic logic unit (ALU), a digital signalprocessor, a microcomputer, a field programmable gate array (FPGA), aSystem-on-Chip (SoC), a programmable logic unit, a microprocessor, orany other device capable of responding to and executing instructions ina defined manner. Portions of the example embodiments and correspondingdetailed description may be presented in terms of software, oralgorithms and symbolic representations of operation on data bits withina computer memory. These descriptions and representations are the onesby which those of ordinary skill in the art effectively convey thesubstance of their work to others of ordinary skill in the art. Analgorithm, as the term is used here, and as it is used generally, isconceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of optical, electrical, or magnetic signals capable of beingstored, transferred, combined, compared, and otherwise manipulated. Ithas proven convenient at times, principally for reasons of common usage,to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, or as is apparent from the discussion,terms such as “processing” or “computing” or “calculating” or“determining” of “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computingdevice/hardware, that manipulates and transforms data represented asphysical, electronic quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

In this application, including the definitions below, the term ‘module’or the term ‘controller’ may be replaced with the term ‘circuit.’ Theterm ‘module’ may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

Software may include a computer program, program code, instructions, orsome combination thereof, for independently or collectively instructingor configuring a hardware device to operate as desired. The computerprogram and/or program code may include program or computer-readableinstructions, software components, software modules, data files, datastructures, and/or the like, capable of being implemented by one or morehardware devices, such as one or more of the hardware devices mentionedabove. Examples of program code include both machine code produced by acompiler and higher level program code that is executed using aninterpreter.

For example, when a hardware device is a computer processing device(e.g., a processor, Central Processing Unit (CPU), a controller, anarithmetic logic unit (ALU), a digital signal processor, amicrocomputer, a microprocessor, etc.), the computer processing devicemay be configured to carry out program code by performing arithmetical,logical, and input/output operations, according to the program code.Once the program code is loaded into a computer processing device, thecomputer processing device may be programmed to perform the programcode, thereby transforming the computer processing device into a specialpurpose computer processing device. In a more specific example, when theprogram code is loaded into a processor, the processor becomesprogrammed to perform the program code and operations correspondingthereto, thereby transforming the processor into a special purposeprocessor.

Software and/or data may be embodied permanently or temporarily in anytype of machine, component, physical or virtual equipment, or computerstorage medium or device, capable of providing instructions or data to,or being interpreted by, a hardware device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. In particular, forexample, software and data may be stored by one or more computerreadable recording mediums, including the tangible or non-transitorycomputer-readable storage media discussed herein.

Even further, any of the disclosed methods may be embodied in the formof a program or software. The program or software may be stored on anon-transitory computer readable medium and is adapted to perform anyone of the aforementioned methods when run on a computer device (adevice including a processor). Thus, the non-transitory, tangiblecomputer readable medium, is adapted to store information and is adaptedto interact with a data processing facility or computer device toexecute the program of any of the above mentioned embodiments and/or toperform the method of any of the above mentioned embodiments.

Example embodiments may be described with reference to acts and symbolicrepresentations of operations (e.g., in the form of flow charts, flowdiagrams, data flow diagrams, structure diagrams, block diagrams, etc.)that may be implemented in conjunction with units and/or devicesdiscussed in more detail below. Although discussed in a particularlymanner, a function or operation specified in a specific block may beperformed differently from the flow specified in a flowchart, flowdiagram, etc. For example, functions or operations illustrated as beingperformed serially in two consecutive blocks may actually be performedsimultaneously, or in some cases be performed in reverse order.

According to one or more example embodiments, computer processingdevices may be described as including various functional units thatperform various operations and/or functions to increase the clarity ofthe description. However, computer processing devices are not intendedto be limited to these functional units. For example, in one or moreexample embodiments, the various operations and/or functions of thefunctional units may be performed by other ones of the functional units.Further, the computer processing devices may perform the operationsand/or functions of the various functional units without sub-dividingthe operations and/or functions of the computer processing units intothese various functional units.

Units and/or devices according to one or more example embodiments mayalso include one or more storage devices. The one or more storagedevices may be tangible or non-transitory computer-readable storagemedia, such as random access memory (RAM), read only memory (ROM), apermanent mass storage device (such as a disk drive), solid state (e.g.,NAND flash) device, and/or any other like data storage mechanism capableof storing and recording data. The one or more storage devices may beconfigured to store computer programs, program code, instructions, orsome combination thereof, for one or more operating systems and/or forimplementing the example embodiments described herein. The computerprograms, program code, instructions, or some combination thereof, mayalso be loaded from a separate computer readable storage medium into theone or more storage devices and/or one or more computer processingdevices using a drive mechanism. Such separate computer readable storagemedium may include a Universal Serial Bus (USB) flash drive, a memorystick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other likecomputer readable storage media. The computer programs, program code,instructions, or some combination thereof, may be loaded into the one ormore storage devices and/or the one or more computer processing devicesfrom a remote data storage device via a network interface, rather thanvia a local computer readable storage medium. Additionally, the computerprograms, program code, instructions, or some combination thereof, maybe loaded into the one or more storage devices and/or the one or moreprocessors from a remote computing system that is configured to transferand/or distribute the computer programs, program code, instructions, orsome combination thereof, over a network. The remote computing systemmay transfer and/or distribute the computer programs, program code,instructions, or some combination thereof, via a wired interface, an airinterface, and/or any other like medium.

The one or more hardware devices, the one or more storage devices,and/or the computer programs, program code, instructions, or somecombination thereof, may be specially designed and constructed for thepurposes of the example embodiments, or they may be known devices thatare altered and/or modified for the purposes of example embodiments.

A hardware device, such as a computer processing device, may run anoperating system (OS) and one or more software applications that run onthe OS. The computer processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For simplicity, one or more example embodiments may beexemplified as a computer processing device or processor; however, oneskilled in the art will appreciate that a hardware device may includemultiple processing elements or processors and multiple types ofprocessing elements or processors. For example, a hardware device mayinclude multiple processors or a processor and a controller. Inaddition, other processing configurations are possible, such as parallelprocessors.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium (memory).The computer programs may also include or rely on stored data. Thecomputer programs may encompass a basic input/output system (BIOS) thatinteracts with hardware of the special purpose computer, device driversthat interact with particular devices of the special purpose computer,one or more operating systems, user applications, background services,background applications, etc. As such, the one or more processors may beconfigured to execute the processor executable instructions.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language) or XML (extensible markuplanguage), (ii) assembly code, (iii) object code generated from sourcecode by a compiler, (iv) source code for execution by an interpreter,(v) source code for compilation and execution by a just-in-timecompiler, etc. As examples only, source code may be written using syntaxfrom languages including C, C++, C#, Objective-C, Haskell, Go, SQL, R,Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5,Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang,Ruby, Flash®, Visual Basic®, Lua, and Python®.

Further, at least one embodiment of the invention relates to thenon-transitory computer-readable storage medium including electronicallyreadable control information (procesor executable instructions) storedthereon, configured in such that when the storage medium is used in acontroller of a device, at least one embodiment of the method may becarried out.

The computer readable medium or storage medium may be a built-in mediuminstalled inside a computer device main body or a removable mediumarranged so that it can be separated from the computer device main body.The term computer-readable medium, as used herein, does not encompasstransitory electrical or electromagnetic signals propagating through amedium (such as on a carrier wave); the term computer-readable medium istherefore considered tangible and non-transitory. Non-limiting examplesof the non-transitory computer-readable medium include, but are notlimited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of the non-transitory computer-readable medium include, but arenot limited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

Although described with reference to specific examples and drawings,modifications, additions and substitutions of example embodiments may bevariously made according to the description by those of ordinary skillin the art. For example, the described techniques may be performed in anorder different with that of the methods described, and/or componentssuch as the described system, architecture, devices, circuit, and thelike, may be connected or combined to be different from theabove-described methods, or results may be appropriately achieved byother components or equivalents.

According to a first embodiment, the present invention refers to acomputer-implemented method (i.e. executed on a processing unit of acomputer, which may e.g. be deployed in an image acquisition orprocessing apparatus) for normalizing medical images. In a preparationphase, a type of image acquisition device may be set for which themethod should be applied (e.g. X-ray, MRI, CT etc.). Thus, the imagesstem from a predetermined type of image acquisition device (in themedical domain also called ‘modality’). The method uses a machinelearning unit and comprises:

receiving a set of image data with images, in particular medical imagesfrom the predetermined type of image acquisition device, at adecomposition unit, wherein the image data have been generated by beingconverted from detector signals, acquired at a detector of an imageacquisition device (e.g. medical modality), wherein the detector signalsare converted by using or applying different settings or configurationsof the image acquisition device-specific processing algorithms;

at a decomposition unit: Decomposing each of the images of the set ofimages into components by incorporating at least information from thedifferent settings of the (applied) image acquisition device-specificprocessing algorithms; and

at a normalizing unit: Normalizing each of the components via a machinelearning unit by processing at least information from the differentsettings of the image acquisition device-specific processing algorithmsto provide a set of normalized images as output with a decreasedvariability score.

Thus, the normalization unit provides normalized images as output whichare normalized for all the different image impressions. The differentimage impressions (image renderings) are due to applying differentconfigurations or settings for signal processing at the acquisitiondevice or modality. In particular, a user, e.g. a radiologist may tuneor set his own preferences (configurations) for detector signalprocessing. It is important to note, that the term “image impressions”does not relate to cognitive image impressions of a user, but tohardcoded settings or configurations for raw detector signal processing,namely to configurations which are used for converting the raw detectorsignals into image data. In each image to be rendered, theseconfigurations or settings are kind of hardcoded or burnt-in. Usually,these configurations are configured one time after first commissioningof the image acquisition device. The configurations may be set by adedicated configuration software which is only operated forcommissioning of the image acquisition device. Usually, the user oroperator the image acquisition device does not have access to thisconfiguration software tool, which is provided by the manufacturer ofthe devices. The configurations are usually not amended or changedafterwards and during operation of the image acquisition device. Theconfigurations are implicitly coded in the generated images, but are notexplicitly available (e.g. in a data memory). The image impression isthus dependent on these configurations mentioned above, namely of imageacquisition device-specific processing algorithms or types thereof or inparticular of modality specific processing algorithms. The imageimpression may relate to different parameters, comprising contrast,brightness, image sharpness/level of detail etc.

At least one embodiment of the invention is based on the observationthat the variability of the image impressions is very high although thesame type of device or modality is used. This is a major problem forsubsequent image analyses and diagnosis. This information from thesettings or configurations is taken into account for normalizing theimages.

On a very general level, at least one embodiment of the inventionproposes a first normalization block which may be used for subsequentmachine learning algorithms (in short ML algorithms) for diverse tasks.The ML algorithms may be executed subsequently in a second block, inparticular, executed in a processing unit and are based on the resultsof the first normalization block. In this embodiment the systemcomprises the first and the second block, i.e. the normalization and thetask specific ML block as well. However, in another embodiment, it isalso possible to only use the first normalization block. Images areprovided as input to the first block (normalization block) andnormalized images are provided as output of this first block.

As mentioned above, the normalized images may be subject to a furtherprocessing, e.g. by way of a neural network algorithm or any other imageprocessing. Thus, in a preferred embodiment, the computer-implementedmethod may further comprise:

Receiving the normalized components at an input interface of a secondmachine learning unit and executing a second machine learning algorithmwith the normalized components for a pre-defined task, wherein thenormalized components are e.g. entered as feature maps of aconvolutional layer of a neural network.

With other words, parameters of both (normalization and task specific)sequential ML units are jointly trained using the cost function of thesecond ML algorithm. This is referred to as end-to-end training andtypically realized by representing both sequential ML units as oneneural network.

The training data set still comprises the variabilities due to hardwareand/or software settings of the image acquisition device (modality) anddue to patient-specific or anatomical parameters. Thus, only variability3) as mentioned above is reduced (due to configurations or setting inthe modality-specific rendering algorithms).

One key advantage of at least one embodiment of the present invention isthat the quality of subsequent image processing techniques, inparticular machine learning algorithms for image evaluation (withrespect to diseases, to be executed in the second block) is improved.Further, training of the neural networks may be much more efficient,because the variability (as mentioned above as 3) aspect) produced bydifferent image impressions is reduced in the first (normalization)block. This helps to remarkably reduce the amount of training data andleads to a better generalization across various image impressions.

In another preferred embodiment, the (first) machine learning unitand/or a second machine learning unit is a neural network, in particulara deep neural network and/or a convolutional neural network, whichtypically will be deployed as a deep convolutional neural network.

In another preferred embodiment, the step of decomposing comprises apyramid decomposition. One technical advantage of this method is that itrepresents an image as a series of components with reduced entropy. Formore details it is referred to [The Laplacian Pyramid as a Compact ImageCode, P. Burt et al, in: IEEE TRANSACTIONS ON COMMUNICATIONS, VOL.COM-31, NO. 4, APRIL 1983], the entire contents of which are herebyincorporated herein by reference.

In another preferred embodiment, each of the images is decomposed intofrequency bands, resulting in so called band-pass images.

In another preferred embodiment, decomposing is executed by wavelettransformation or by Fourier transformation or by Laplacetransformation.

In another preferred embodiment, normalizing each of the components isexecuted by histogram equalization or mean and variance normalization.

The normalization procedure preferably comprises

1) an image acquisition device-dependent topology based on explicitknowledge of the generation of different image impressions (e.g. thedecomposition of an image into specific frequency bands, which areparameterisably and distorted in the devices) and

2) a set of parameters Φ (e.g. for normalization, see examples below).

The application of such a normalization procedure for therobustification of ML algorithms by explicit reduction of thevariability of different image impressions is a new concept. To definethe topology of the normalization procedure, explicit knowledge aboutthe generation of different image impressions is used.

Thus, a (first) machine learning algorithm (and corresponding neuralnetwork) is used for determining the normalization parameters and asecond or other machine learning algorithm (and corresponding secondneural network) may subsequently be applied for any task which has to beexecuted on the provided normalized images.

Parameters of both networks may be trained in common, i.e. parameters Φof the normalization network (first block) with parameters Ω of thesubsequent task-specific ML algorithm (second block).

There do exist several options for calculating the set of parameters Φfor normalization:

1) Manual determination (e.g. normalization of frequency bands to anaverage value of 0);

2) Independent training of the ML algorithm which is used fornormalization compared to the one for training of the subsequenttask-specific ML algorithm. Independent training means using a separate,own cost function (e.g. with the aim of calculating the mean values offrequency bands to the corresponding mean values of a reference imageimpression);

3) Training all or a subset of the set of parameters Φ for normalizationas part of the task-specific ML algorithm=cost function of the MLalgorithm:

-   -   All or a subset of the parameters Φ and Ω are learned together    -   The resulting information at the output of the normalization is        objective of the ML algorithm is optimized and can be very much        influenced by a deviate from clinically relevant image        impression.

In another preferred embodiment, the method comprises:

Resampling of each of the components, in particular with respect toresolution. For example, if an input image is provided in a very highresolution, for which the decomposition and normalization is derivedfrom the creation of different image impressions, but the subsequentimage processing of the second ML-block does not need this highresolution, resampling may be used. This may help to reduce storage andprocessing capacities.

In another preferred embodiment, a verification algorithm may beexecuted after providing the normalized images (output or result of thefirst block). The verification algorithm may comprise the step ofsuperimposing the normalized components on each of the images therebycreating superimposed images.

In another preferred embodiment, the images represent medical imagesfrom a preset type of image acquisition device (Computer tomography/CT,x-ray, ultrasound, magnetic resonance tomography, etc.). However, alsonon-medical images may be subject to the processing, explained herein,e.g. RGB images.

In another preferred embodiment, normalizing comprises a global and/or alocal normalization by locally normalizing all extracted components on aspatially local manner.

In another preferred embodiment, normalizing the components via themachine learning unit is executed by using image processing knowledge ofthe image and applied image processing techniques. In particular, theprocessing knowledge may comprise knowledge about the image processingsteps which have been applied to convert the detector signals, receivedfrom the image acquisition device's detector to provide an imagerepresentation. Moreover, the processing knowledge may in additioncomprise:

processing anatomical information comprising identification of e.g.anatomical structures (e.g. organ or structure segmentation) in case ofmedical images and/or

processing patient information, comprising relative positioning of thepatient in the image acquisition image acquisition device and/or

processing patient characteristics, i.e. data relating to the patientwhich is examined and represented in the image set, like age, sex,height, weight etc.

processing image acquisition device information, e.g., physical settingslike voltage, emitted photon energy or detector type.

This makes it possible to egalize the image impression between differentimages sets (e.g. image studies, DICOM sets) from a preset imageacquisition device (e.g. radiological images), which have been processedwith different image processing steps (algorithms). For instance,various image impressions for X-ray images can be created by performinga photon energy normalization of the detector information, followed bynoise reduction and gamma filtering for logarithmic scaling, and aparameterized nonlinear weighting of different frequency components. Thelast processing step comprises a decomposition step (e.g., a Laplacianpyramid decomposition, for more details see also [The Laplacian Pyramidas a Compact Image Code, P. Burt et al, in: IEEE TRANSACTIONS ONCOMMUNICATIONS, VOL. COM-31, NO. 4, APRIL 1983]), component-wiseconfigurable nonlinear transformations (which can be adjusted to theradiologist's demands) and a final composition step (e.g., a weightedsuperposition).

In another preferred embodiment, normalizing may use and apply a slidingwindow technique. A sliding window technique relates to an optimizationalgorithm for data processing in order to reduce the time complexity toO(n). An example for local image normalization is the application ofhistogram equalization or mean and variance normalization for smallimage regions, which are extracted by shifting a rectangular window overthe image. One technical advantage of exploiting spatially localtechniques is that the contrast in very light or dark regions can beremarkably enhanced.

In another preferred embodiment, the hyperparameters (e.g. percentile ofhistogram-based equalizations) and/or the parameters of thenormalization are not pre-set, but instead are determined and learned(calculated) by the machine learning algorithm. For example, duringnormalization, particular frequency bands may be selected due to imageacquisition device-specific knowledge of the image processingalgorithms. For example, the amount of the frequency bands to be usedmay represent a hyperparameter of the normalization. This parameter maybe subject of adaption and may be adapted during training of the MLalgorithm.

In another embodiment, the invention relates to a system for normalizingmedical or non-medical images from a predetermined type of imageacquisition device using a machine learning unit. The system iscomputer-based and comprises:

a decomposition unit with an image input interface for receiving a setof image data with images, wherein the image data have been generated bybeing converted from detector signals, acquired at a detector of animage acquisition device, wherein the detector signals are converted byapplying different settings of configurations in the image acquisitiondevice-specific image processing algorithms;

wherein, the decomposition unit, incorporates at least information fromthe different settings in the modality-specific image processingalgorithms and is adapted for decomposing each of the images of the setof images into components; and

a normalizing unit, which is adapted for normalizing each of thecomponents via a machine learning unit by processing at leastinformation from the different settings of the image acquisitiondevice-specific image processing algorithms to provide a set ofnormalized images with a decreased variability score.

In another embodiment, the invention relates to a computer programproduct comprising a computer program, the computer program beingloadable into a memory unit of a computing unit, including program codesections to make the computing unit execute the method for imagenormalization according to an embodiment of the invention, when thecomputer program is executed in the computing unit.

In another embodiment, the invention relates to a computer-readablemedium, on which program code sections of a computer program are storedor saved, the program code sections being loadable into and/orexecutable in a computing unit to make the computing unit execute themethod for image normalization according to an embodiment of theinvention, when the program code sections are executed in the computingunit.

In another embodiment, the invention relates to a computer program, thecomputer program being loadable into a memory unit of a computer, inparticular, of an image processing or image acquisition device.

The realization of embodiments of the invention by a computer programproduct and/or a computer-readable medium has the advantage that alreadyexisting modalities and servers can be easily adopted by softwareupdates in order to work as proposed by embodiments of the invention.

The properties, features and advantages of this invention describedabove, as well as the manner they are achieved, become clearer and moreunderstandable in the light of the following description andembodiments, which will be described in more detail in the context ofthe drawings. This following description does not limit the invention onthe contained embodiments. Same components or parts can be labeled withthe same reference signs in different figures. In general, the figuresare not for scale.

It shall be understood that a preferred embodiment of the presentinvention can also be any combination of the dependent claims or aboveembodiments with the respective independent claim.

Embodiments of the present invention relates to calculating normalizedimages for a set of images, acquired at a image acquisition device of acertain type, e.g. a radiological image acquisition device.

In the following a definition of terms used within this application isgiven.

The image may be a medical image with clinical content. It may be e.g. aDICOM image. The image may be generated by a certain modality as imageacquisition device. The type of modality is not restricted for using thenormalization method. However, the method is dedicated to a specifictype, so that all images, acquired of this modality are re-calculatedfor being provided in a standardized form. Deviations of imageimpressions are equalized. Other embodiments refer to non-medical imagesacquired on non-medical devices, such as cameras.

Equalized impressions relate to a relation between pixel intensities,e.g., by way of contrast, brightness or sharpness.

In a preferred embodiment two different blocks or structures are used:

A normalization block and

A ML block for a certain image task (e.g. classification, regressionetc.)

Both blocks may comprise or access different neural networks, inparticular CNNs, with different machine learning algorithms. All orsubsets of the parameters for normalization may be learned in common—inparticular in one single training—with the parameters for the taskspecific ML algorithm, which may be executed subsequently to providingthe normalized images.

Decomposing is a computer operation via a decomposition algorithm. Thedecomposition algorithm divides the input image in a pre-configurableset of frequency bands. Pyramid decomposition may be used in thisrespect. The result of decomposing is a set of decomposed or band-passimages. The band-pass images may be individually histogram-equalized ormean and variance standardized to account for pixel-value shiftsproduced by non-linear processing in the respective frequency band. Inother embodiments any concept for decomposing an image into band-passcomponents, e.g., based on Fourier/Laplace or wavelet transformation maybe used, too.

Normalizing is a computer operation by executing a normalizingalgorithm. The normalizing algorithm adjusts pixel values of theindividual band-pass images according to a global and localnormalization strategy.

FIG. 1 illustrates the general pipeline for normalizing images. A set ofimages S with images I1 to IV is provided. The images I1 to IV havedifferent image impressions due to processing the same detectorinformation with different configurations of the modality-specific—ormore general: image acquisition device specific—image processingtechnique.

In general, it has to be noted that the difference between images I1 to15 is only produced by variability 3), as mentioned above. The recordeddetector information is processed (converted) to produce a vendor-and/or application-specific image representation. These differences arenot based on other influences, like anatomical variabilities.

If the images represent medical images the image impression can differdue to the shape of the anatomical structures which vary with therelative positioning and characteristics (e.g., age, gender) of apatient. It is important to note, that the provided images are createdfrom the same detector information. There is no variability due topositioning or patient characteristics. The term “image impression” sofar refers to different configurations of the image processing pipeline(on the related devices and apparatuses) to convert the same detectorinformation to different image renderings, adjusted to the radiologists'needs.

Each of the images I1 to IV is decomposed in a decomposition unit Dthereby creating components C1 to Cn. Each of the components C1 to Cn isnormally resampled thereby creating resampled components CR1 to CRnwhich represent input for the Machine Learning unit ML. Resampling mightbe important for images (e.g. X-ray) with a high resolution. Images witha resolution of 3000×3000 pixel can e.g. resampled to a resolution of1000×1000 pixel. The reduction of the resolution might be relevant formachine learning units ML which for the sake of processing efficiencywork with lower resolution images. The use of the resampled componentsCR1 to CRn instead of the complete image (e.g. downsampling), as knownfrom the state of the art, improves and increases the training speed.Performing the decomposition and normalization before making thedownsampling is motivated by the fact that explicit knowledge on thecreation of image impressions (in the device for high-res images) shouldbe incorporated. This is a major advantage (training speed) over priorart.

The normalization results in that the image impressions (e.g. contrast,intensity) are more homogeneous but still different enough in order toachieve good training results by the machine learning unit ML.Generally, the goal of the normalization is to reduce the variability 3)produced by different image impressions. This means that the normalizedimage still include the variabilities 1) and 2) obtained due to patientcharacteristics (e.g., age, gender), patient positioning (e.g., sitting,standing) and device setting (e.g., voltage, emitted photon energy).These variabilities 1) and 2) are diverse enough to achieve goodtraining results.

The machine learning unit ML may be adapted to receive a set ofdecomposed images, i.e. a set of normalized image components. Astraightforward approach is to exploit three normalized image componentsas input channels of a deep convolutional neural network which wastopologically optimized for RGB images. Alternatively, a flexible numberof normalized components can be directly used as input feature maps of adeep convolutional neural network.

Generally, the system SYS is adapted for image normalization. The systemSYS may be implemented on an image acquisition device or on a server indata network connection with the image acquisition device. The systemSYS at least uses one neural network, in particular a CNN fordetermining the normalization parameters, which are used for normalizingthe images I. After having calculated the set of normalized images CN, avariety of task specific additional image data processings may becarried out on the normalized images, like those which may beimplemented by a second neural network or by executing second machinelearning algorithms ML2.

FIG. 2 illustrates a specific embodiment for the application of thegeneral pipeline for normalizing images according to FIG. 1 . A set ofimages S with medical images I1 to IV is provided. The images I1 to IVhave different image impressions created by processing the same detectorinformation with different configurations of the modality-specific orimage acquisition device-specific image processing technique.

In image processing different physical parameters (signals at thedetector of the modality) are transformed or converted into imagerepresentations with possibly different image impressions (e.g.contrast, intensity) by configurable processing units. For instance, inX-ray apparatuses the measured photon energy is converted into aninterpretable X-ray image. The doctors can adjust the image impressionaccording to device settings (contrast, brightness etc.). Thisindividual adjustment can be done by weighting components of frequencybands with parameterisable non-linearities. The huge advantage of thenormalization procedure of the current invention consists in the reducedinfluence of the adjustable parameters specifying the non-linearities inthe different frequency bands. The fundamental goal is to get the sameimage rendering result for different parameter settings, i.e. to unifyover the image impressions which are created by changing the band-wiseparameters of the nonlinearities.

The images I1 to IV differ due to the processing of the same detectorinformation with different configurations of the modality-specific imageprocessing technique or thus—more general—to image acquisitiondevice-specific processing algorithms and settings.

Each of the images I1 to IV is decomposed in a decomposition unit Dthereby creating components C1 to Cn.

In a preferred embodiment, the decomposition is carried out by pyramiddecomposition. Each of the input images I1 to IV is decomposed into ninecomponents C1 to C9 representing nine frequency bands. The resultingband-pass images are normalized in that each of the components C1 to C9are individually histogram-equalized to account to pixel shifts producedby nonlinear processing in the respective frequency band. The normalizedcomponents CN1 to CN9 are provided separately to the machine learningunit ML2. This is done without combining the normalized components CN1to CN9 into one medical image. The advantage of this approach is that itallows the machine learning unit ML2 to exploit the information providedby the normalized image components in a more flexible way. The machinelearning unit ML2 executes the training with the normalized imagecomponents. Training results are improved in that unpredictable resultsdue to mismatches between training and testing conditions produced byvarious image impressions are mitigated.

Possible alternatives to pyramid decomposition with task-specific numberof frequency bands are, e.g. configurable decompositions based on thewavelet, Fourier or Laplacian transformation.

Alternative solutions exist for normalization. Besides adjusting thepixel values of the individual band-pass images globally (i.e., based onthe histogram of the entire band-pass image, e.g., by way of histogramequalization or mean and variance normalization), the band-pass imagescould be normalized on a spatially local manner. For instance, byexploiting the identification of anatomical structures or by using asliding window as similarly applied for adaptive histogram equalization.Independent of the dedicated normalization concept, an alternative tomanually set the respective hyperparameters is to represent thenormalization as part of the processing carried out by the machinelearning unit ML2 and thus to learn them in an end-to-end way (i.e.,during the training phase).

The machine learning unit ML, ML2 is preferably a neural network NN. Ina specific embodiment a neural network is a deep neural network withmore than two layers. In a further embodiment the normalized componentsCN1 to CNn are input as feature maps of a convolutional layer of thedeep neural network. This further improves the training results.

FIG. 3 shows example images for four different image impressions without(upper row) and with (lower row) applying the normalization procedureaccording to the invention. The nine histogram normalized frequencycomponents CN1 to CNn of FIG. 2 are superimposed. By comparing theoriginal images (upper row) with the respective normalized images (lowerrow), it becomes obvious that the inventive normalization method unifiesthe image appearance.

FIG. 4 shows a flow chart with a sequence of method steps. In step S1 aset of image data with images is received at a decomposition unit or atanother unit and provided to the decomposition unit. In step S2 thereceived image is decomposed for providing decomposed or band passimages. In step S3 the decomposed images are normalized. After this, themethod may end or may optionally be extended by step S4, which relatesto resampling the normalized images. Optionally, a verificationalgorithm may be executed on the result dataset with the normalizedimages.

A single unit or device may fulfil the functions of several itemsrecited in the claims. The mere fact that certain measures are recitedin mutually different dependent claims does not indicate that acombination of these measures cannot be used to advantage.

A computer program may be stored/distributed on a suitable medium, suchas an optical storage medium or a solid-state medium, supplied togetherwith or as part of other hardware, but may also be distributed in otherforms, such as via the Internet or other wired or wirelesstelecommunication systems.

Embodiments of the present invention has been described in the generalcontext of computer-executable instructions, such as program modules,being executed by a personal computer. However, the methods ofembodiments of the present invention may be effected by other apparatus.Program modules may include routines, programs, objects, components,data structures, etc. that perform a task(s) or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat at least some aspects of the present invention may be practicedwith other configurations, including hand-held devices, multiprocessorsystems, microprocessor-based or programmable consumer electronics,network computers, minicomputers, set top boxes, mainframe computers,and the like. At least some aspects of the present invention may also bepracticed in distributed computing environments where tasks areperformed by remote processing devices linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in local and/or remote memory storage devices.

Any reference signs in the claims should not be construed as limitingthe scope.

Wherever not already described explicitly, individual embodiments, ortheir individual aspects and features, described in relation to thedrawings can be combined or exchanged with one another without limitingor widening the scope of the described invention, whenever such acombination or exchange is meaningful and in the sense of thisinvention. Advantages which are described with respect to a particularembodiment of present invention or with respect to a particular figureare, wherever applicable, also advantages of other embodiments of thepresent invention.

The patent claims of the application are formulation proposals withoutprejudice for obtaining more extensive patent protection. The applicantreserves the right to claim even further combinations of featurespreviously disclosed only in the description and/or drawings.

References back that are used in dependent claims indicate the furtherembodiment of the subject matter of the main claim by way of thefeatures of the respective dependent claim; they should not beunderstood as dispensing with obtaining independent protection of thesubject matter for the combinations of features in the referred-backdependent claims. Furthermore, with regard to interpreting the claims,where a feature is concretized in more specific detail in a subordinateclaim, it should be assumed that such a restriction is not present inthe respective preceding claims.

Since the subject matter of the dependent claims in relation to theprior art on the priority date may form separate and independentinventions, the applicant reserves the right to make them the subjectmatter of independent claims or divisional declarations. They mayfurthermore also contain independent inventions which have aconfiguration that is independent of the subject matters of thepreceding dependent claims.

None of the elements recited in the claims are intended to be ameans-plus-function element within the meaning of 35 U.S.C. § 112(f)unless an element is expressly recited using the phrase “means for” or,in the case of a method claim, using the phrases “operation for” or“step for.”

Example embodiments being thus described, it will be obvious that thesame may be varied in many ways. Such variations are not to be regardedas a departure from the spirit and scope of the present invention, andall such modifications as would be obvious to one skilled in the art areintended to be included within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method for normalizingimages from a type of image acquisition device, the computer-implementedmethod comprising: receiving, at a decomposition unit, a set of imagedata with images, the image data having been generated by beingconverted from detector signals acquired at a detector of an imageacquisition device, wherein the detector signals were converted byapplying different settings of image acquisition device-specificprocessing algorithms; decomposing, at the decomposition unit, each ofthe images of the set of image data into components by incorporating atleast information from the different settings of the image acquisitiondevice-specific processing algorithms; normalizing, at a normalizingunit, each of the components via a first machine learning unit toproduce normalized components, by processing at least information fromthe different settings of the image acquisition device-specificprocessing algorithms to provide a set of normalized images with adecreased variability score; and receiving the normalized components atan input interface of a second machine learning unit and executing amachine learning algorithm with the normalized components for a definedtask, wherein the normalized components are entered as feature maps of aconvolutional layer of a neural network, and during training of thefirst machine learning unit and the second machine learning unit, atleast a part of a plurality of parameters of the first machine learningunit are jointly trained with parameters used and processed in thesecond machine learning unit using a same cost function.
 2. Thecomputer-implemented method of claim 1, wherein at least one of thefirst machine learning unit or the second machine learning unit is aneural network.
 3. The computer-implemented method of claim 1, whereindecomposing comprises a pyramid decomposition.
 4. Thecomputer-implemented method of claim 1, wherein each of the images isdecomposed into frequency bands, resulting in band-pass images.
 5. Thecomputer-implemented method of claim 1, wherein the decomposing isexecuted by wavelet transformation or by Fourier transformation or byLaplace transformation.
 6. The computer-implemented method of claim 1,wherein the normalizing of each of the components is executed byhistogram equalization.
 7. The computer-implemented method of claim 1,further comprising: resampling each of the components with respect toresolution.
 8. The computer-implemented method of claim 4, wherein thenormalizing includes a global normalization and a local normalization onthe components by locally normalizing all components in a spatiallylocal manner.
 9. The computer-implemented method of claim 1, wherein thenormalizing of the components via the first machine learning unit isexecuted by at least one of processing anatomical information includingidentification of anatomical structures in the images, or processing atleast one of patient information or image acquisition device specificinformation, including patient characteristics and relative positioningof a patient in the image acquisition device.
 10. Thecomputer-implemented method of claim 1, wherein the normalizing includesusing a sliding window.
 11. A system for normalizing medical images froma type of image acquisition device, the system comprising: adecomposition unit including an image input interface to receive a setof image data with images, the image data having been generated by beingconverted from detector signals, acquired at a detector of an imageacquisition device, the detector signals having been converted byapplying different settings of image acquisition device-specificprocessing algorithms, wherein the decomposition unit is configured todecompose each of the images of the set of image data into components byincorporating at least information from the different settings of theimage acquisition device-specific processing algorithms; and anormalizing unit configured to normalize each of the components via afirst machine learning unit to produce normalized components, byprocessing at least information from the different settings of the imageacquisition device-specific processing algorithms to provide a set ofnormalized images with a decreased variability score; wherein thenormalized components are received at an input interface of a secondmachine learning unit and a machine learning algorithm is executed withthe normalized components for a defined task, the normalized componentsare entered as feature maps of a convolutional layer of a neuralnetwork, and during training of the first machine learning unit and thesecond machine learning unit, at least a part of a plurality ofparameters of the first machine learning unit are jointly trained withparameters used and processed in the second machine learning unit usinga same cost function.
 12. A non-transitory computer program productstoring a computer program, the computer program being loadable into amemory unit of a computing unit and including program code sections tocause the computing unit to, when the computer program is executed inthe computing unit, perform a method comprising: receiving a set ofimage data with images, the image data having been generated by beingconverted from detector signals acquired at a detector of an imageacquisition device, wherein the detector signals were converted byapplying different settings of image acquisition device-specificprocessing algorithms; decomposing each of the images of the set ofimage data into components by incorporating at least information fromthe different settings of the image acquisition device-specificprocessing algorithms; normalizing each of the components via a firstmachine learning unit to produce normalized components, by processing atleast information from the different settings of the image acquisitiondevice-specific processing algorithms to provide a set of normalizedimages with a decreased variability score; and receiving the normalizedcomponents at an input interface of a second machine learning unit andexecuting a machine learning algorithm with the normalized componentsfor a defined task, wherein the normalized components are entered asfeature maps of a convolutional layer of a neural network, and duringtraining of the first machine learning unit and the second machinelearning unit, at least a part of a plurality of parameters of the firstmachine learning unit are jointly trained with parameters used andprocessed in the second machine learning unit using a same costfunction.
 13. An image acquisition device, comprising: a system fornormalizing medical images from a type of image acquisition device, thesystem including a decomposition unit including an image input interfaceto receive a set of image data with images, the image data having beengenerated by being converted from detector signals, acquired at adetector of an image acquisition device, the detector signals havingbeen converted by applying different settings of image acquisitiondevice-specific processing algorithms, wherein the decomposition unit isconfigured to decompose each of the images of the set of image data intocomponents by incorporating at least information from the differentsettings of the image acquisition device-specific processing algorithms,and a normalizing unit configured to normalize each of the componentsvia a first machine learning unit to produce normalized components, byprocessing at least information from the different settings of the imageacquisition device-specific processing algorithms to provide a set ofnormalized images with a decreased variability score, wherein thenormalized components are received at an input interface of a secondmachine learning unit and a machine learning algorithm is executed withthe normalized components for a defined task, the normalized componentsare entered as feature maps of a convolutional layer of a neuralnetwork, and during training of the first machine learning unit and thesecond machine learning unit, at least a part of a plurality ofparameters of the first machine learning unit are jointly trained withparameters used and processed in the second machine learning unit usinga same cost function.
 14. The computer-implemented method of claim 1,wherein the first machine learning unit is a neural network.
 15. Thecomputer-implemented method of claim 2, wherein the neural network is adeep neural network or a deep convolutional neural network.
 16. Thecomputer-implemented method of claim 14, wherein the neural network is adeep neural network or a deep convolutional neural network.
 17. Thecomputer-implemented method of claim 2, wherein the decomposing isexecuted by wavelet transformation, by Fourier transformation, or byLaplace transformation.
 18. The computer-implemented method of claim 2,wherein the normalizing of each of the components is executed byhistogram equalization.